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Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 
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•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
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Abstract 

This  paper  addresses  the  need  for  predicting  performance  in  a  system  of  systems  (SoS) 
during  incremental  development  and  for  dealing  with  the  inherent  variability  associated  with 
predicting  performance.  Historically,  senior  decision-makers  have  used  technical 
performance  measures  (TPM),  along  with  modeling  and  simulation,  to  predict  whether  a 
system  under  development  will  meet  performance  requirements.  This  methodology  does  not 
appear  to  be  directly  translatable  to  SoS  for  several  reasons,  including  the  inherent 
complexity  of  the  SoS  and  the  operational  flexibility  the  end  user  may  have  in  employing  the 
SoS.  An  approach  for  dealing  with  the  SoS  performance  prediction  has  been  presented 
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previously.  It  laid  out  a  notional  approach  to  dealing  with  this  issue.  This  approach  has  been 
generalized  to  address  the  use  and  integration  of  multiple  technologies  into  an  SoS  and  into 
the  decision-maker’s  options  in  the  use  of  these  technologies  that  is  rooted  in  using  subject 
matter  expert  input  and  historical  data.  This  methodology  is  used  to  develop  a  metric  defined 
as  an  SoS  performance  measure  (SPM),  which  serves  as  an  equivalent  in  functionality  to  a 
TPM  for  a  SoS.  Similar  to  TPMs,  an  approach  to  developing  tolerance  bands  is  presented  to 
be  used  for  predicting  the  status  of  development  as  a  function  of  time.  The  methodology  is 
first  presented  as  a  deterministic  method  for  predicting  SoS  performance  during 
development.  This  method  is  then  demonstrated  using  an  example  case  to  illustrate  the 
methodology.  However,  many  of  the  component  variables  have  significant  uncertainty 
associated  with  them  during  SoS  development  and  integration  into  the  SoS.  The  paper 
provides  an  approach  for  expanding  the  SPM  concept  to  account  for  this  uncertainty  using  a 
stochastic  approach  to  address  this  issue. 

A  Constraining  Environment  Within  Defense  Acquisition 

As  one  looks  over  the  span  of  human  history,  one  can  see  that  human  society  has 
transitioned  from  a  simple  hunter-gather  society  to  basic  communal  organizations,  to  the 
nation  states,  and  to  the  international  organizations  of  today.  Directly  related  to  the  growth  of 
mankind’s  increased  societal  relationships  has  been  the  increasingly  complex  infrastructure 
and  systems  that  are  required  to  support  and  enable  those  relationships.  Engineering,  as  a 
field  of  endeavor,  has  had  a  role  solving  the  problems  that  this  increase  in  complexity  has 
created.  This  relationship  goes  back  to  the  earliest  known  civil  engineer,  Imhotep,  who  is 
credited  with  the  design  and  management  of  the  construction  of  the  Pyramid  of  Djoser 
(“Engineering,”  2012).  As  can  be  seen  in  Figure  1,  the  rate  of  complexity  in  systems  has 
increased  rapidly  since  the  start  of  the  Industrial  Age.  Since  World  War  II,  the  growth  has 
been  almost  exponential,  so  much  so  that  the  21st  century  is  being  referred  to  as  “The 
Systems  Century”  (Tetlay  &  John,  2009). 


Figure  1.  Growth  of  System  Complexity 
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The  emergence  of  systems  engineering  as  a  field  has  helped  in  controlling  the 
effective  development  of  complex  systems,  but,  as  Smith  and  Meyers  (2008)  noted,  “large, 
complex  systems  development  has  always  been  challenging,  even  when  the  ‘only’  thing  a 
program  manager  had  to  worry  about  were  cost,  schedule,  and  performance  within  a  single 
program.”  Today,  just  as  the  interaction  and  exchange  of  information  within  societies 
continues  to  increase,  system  complexity  and  interdependence  continues  to  increase. 
Fielded  capabilities  are  now  often  comprised  of  multiple  independent  systems  working 
together  to  provide  specific  sets  of  capabilities  to  the  end  user.  This  is  leading  to  the 
development  of  a  new  field  of  knowledge  within  systems  engineering  that  focuses  on  the 
technical  understanding  and  management  of  complex  systems  of  systems  (SoS).  This  is 
especially  true  within  the  Department  of  Defense  (DoD),  where  SoS  have  emerged  as  the 
preferred  approach  for  the  acquisition  of  capabilities.  This  shift  is  driving  the  need  for  new 
tools  to  aid  the  SoS  program  manager  (PM)/system  engineer,  but  let  us  first  understand  the 
DoD  definition  of  an  SoS  and  why  SoS  are  DoD’s  preferred  way  of  acquiring  capability. 

Drivers  of  SoS  Development  Within  Defense  Acquisition 

The  U.S.  defense  acquisition  system  is  operating  in  an  environment  of  increasing 
fiscal  constraint  while  being  challenged  to  meet  increasingly  complex,  adaptive,  and  capable 
threats  (DoD,  2011).  The  overall  trend  of  DoD  budgets  has  been  declining  as  a  percentage 
of  gross  domestic  product  (GDP),  falling  from  approximately  15%  in  the  mid-1950s  to 
around  4%  today.  As  can  be  seen  in  Figure  2,  the  Congressional  Budget  Office  (CBO) 
anticipates  that  the  portion  of  GDP  dedicated  to  the  DoD  will  continue  to  decrease  over  the 
next  several  decades.  This  will  intensify  the  motivation  to  obtain  greater  utility  from  systems 
developed  and  procured.  Additionally,  while  the  overall  DoD  budget  decreased,  the  amount 
available  for  systems  development  and  acquisition  is  expected  to  be  further  reduced  as 
operations,  maintenance,  and  personnel  costs  are  expected  to  increase  within  the 
constrained  budget. 


Figure  2.  Costs  of  DoD  Plans  as  a  Share  of  Domestic  Output 

In  addition  to  the  fiscal  constraints  facing  the  DoD,  the  areas  where  the  U.S.  has 
historically  maintained  advantages  due  to  its  technological  edge  are  decreasing  (Davis  & 
Wilson,  201 1 ).  This  means  that  asymmetrical  warfare  options  are  increasing  as  the  cost  to 
obtain  and  implement  the  enabling  technologies  decrease.  Against  this  environment,  the 
present  U.S.  general  force  structure  and  its  underlying  operational  concepts  are  becoming 
obsolete  (Krepinevich,  2009).  To  address  these  issues,  the  DoD  has  started  to  shift  away 
from  traditional  system-based  acquisitions  towards  capability-based  acquisitions  (Chairman 
of  the  Joint  Chiefs  of  Staff  [CJCS],  2007).  This  shift  seeks  to  better  leverage  existing 
investments  in  procurement  systems  while  providing  the  DoD  with  increased  capabilities  and 
performance,  which,  as  a  result,  inherently  moves  the  DoD  towards  an  SoS-based 
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acquisition  approach.  Recognizing  this,  the  DoD  developed  the  Systems  Engineering  Guide 
(Office  of  the  Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Systems 
and  Software  Engineering  [ODUSD(A&T)  SSE],  2008)  to  help  defense  acquisition  programs 
understand  the  emerging  SoS  concept  and  to  aid  in  achieving  program  success. 

It  should  be  recognized  that  SoSs  are  not  new  to  the  DoD.  Even  before  the  term 
came  into  popular  use,  the  DoD  and  the  National  Aeronautics  and  Space  Administration 
(NASA)  were  developing  space  systems  that  required  coordination  among  the  launch 
systems,  payload  vehicles,  and  their  telemetry,  control,  and  analysis  systems.  Air  defense 
and  later  ballistic  missile  defense  systems  drove  the  need  to  integrate  sensor,  weapon,  and 
command  and  control  data.  The  DoD  is  striving  to  ensure  that  all  procured  systems  operate 
more  effectively  together  in  order  to  achieve  greater  operational  capability,  which  is  reflected 
in  the  shift  away  from  system-level  performance  requirements  and  toward  the  development 
and  implementation  of  capability-based  requirements. 

Definition  and  Application  of  SoS  Within  Defense  Acquisition 

The  field  of  SoS  engineering,  development,  integration,  sustainment,  and 
management  requires  the  decision-maker  to  face  both  the  traditional  challenges  associated 
with  any  complex  system  (Jamshidi,  2006)  and  the  additional  challenges  associated  with 
having  to  analyze,  organize,  and  integrate  the  constituent  systems  (existing  and 
developmental)  into  an  integrated  SoS  capability.  Within  the  larger  technical  community,  the 
definition  of  what  an  SoS  is  varies  significantly  and  is  often  problem  specific  (Eisner,  1993; 
Shenhar,  1994;  Manthorpe,  1996;  Maier,  1998;  Kotov,  1997;  Krygiel,  1999;  Keating  et  al., 

2003).  Within  the  DoD,  the  U.S.  Defense  Acquisition  Guide  (DoD,  201 1 )  defines  an  SoS  as 
a  set  or  arrangement  of  systems  that  results  from  independent  systems  integrated  into  a 
larger  system  that  delivers  unique  capability.  Based  on  work  done  by  Maier  (1998)  and 
Dahmann  and  Balwin  (2008),  the  DoD  has  defined  four  types  of  SoS:  virtual,  collaborative, 
acknowledged,  and  directed.  Within  U.S.  defense  acquisition,  the  most  common  SoS  type  is 
the  acknowledged  SoS.  For  this  reason,  understanding  issues  related  to  acknowledged  SoS 
and  developing  technical  and  programmatic  management  tools  supporting  them  is  critical  to 
enabling  DoD  program  success.  An  acknowledged  SoS  is  defined  as  one  where  there  exists 
acknowledged  and  documented  capability  requirements  and  funding,  and  a  designated  lead 
is  funded,  but  the  individual  constituent  systems  retain  their  independent  developmental, 
performance,  funding,  and  reporting  paths  (ODUSD[A&T]  SSE,  2008).  The  objective  of  an 
acknowledged  SoS  is  to  successfully  integrate  the  capabilities  of  constituent  systems  to 
provide  required  capabilities  that  the  constituent  systems  cannot  achieve  independently. 

This  ability  to  integrate  ongoing  developmental  efforts  with  fielded  systems  to  improve 
warfighting  capability  is  one  of  the  reasons  that  the  DoD  is  seeing  a  continual  increase  in  the 
number  of  acknowledged  SoSs  being  developed.  Although  the  SoS  acquisition  process  has 
the  potential  to  achieve  this  objective,  monitoring  the  progress  of  an  acknowledged  SoS  can 
often  be  problematic,  as  the  individual  component  systems  retain  their  independent 
developmental,  funding,  development,  and  sustainment  approaches.  Overall,  effective  SoS 
technical  and  programmatic  management  methodologies  are  still  evolving  to  sufficiently 
address  SoS-specific  technical  and  programmatic  issues. 

System  Versus  SoS  Development  Tools,  Metrics,  and  Methodologies 

One  of  the  recognized  advantages  of  an  SoS  is  that  enhanced  performance  is 
achieved  by  the  integration  of  the  individual  independent  and  useful  systems  into  an 
integrated  SoS  that  delivers  unique  capabilities.  These  unique  capabilities  are  often 
achieved  through  changes  to  the  operational  mission,  maintenance  and  support,  or 
employment  methodology  of  the  individual  system  in  the  SoS.  For  example,  a  system 
originally  developed  for  airborne  use,  where  weight  is  often  a  critical  factor,  may  be 
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reconfigured  for  use  on  an  unmanned  vehicle  (UV)  in  an  SoS.  In  this  application,  weight 
may  no  longer  be  a  critical  factor  for  the  SoS,  but  reliability  may  be.  As  airborne  systems  are 
generally  used  on  missions  of  relatively  short  durations,  what  was  acceptable  reliability 
before  for  the  system  may  not  be  operationally  effective  for  a  long  duration  UV  mission. 

Thus,  the  carryover  and  monitoring  of  the  existing  system  level  data  and  metrics  from  the 
constituent  systems  could  result  in  an  erroneous  view  of  the  SoS  and  its  capabilities. 

Additionally,  as  constituent  systems  are  integrated  into  the  SoS,  this  may  result  in  some  of 
the  initial  capabilities  of  the  individual  systems  being  either  enhanced  (such  as  interfacing  an 
improved  sensor  to  an  existing  detection  system)  or  degraded  (removing  a  data  input 
previously  provided  by  another  interfacing  system  that  is  not  included  within  the  SoS 
capability  set).  These  factors  contribute  to  the  ineffectiveness  of  many  system-level  metrics 
supporting  accurate  prediction  of  SoS  status.  Some  of  these  metrics  and  the  approaches 
being  used  to  address  their  use  within  an  SoS  are  discussed  as  follows. 

As  noted  by  the  SoS  Systems  Engineering  Guide  (ODUSD[A&T]  SSE,  2008), 

The  architecture  of  an  SoS  addresses  the  concept  of  operations  for  the  SoS 
and  encompasses  the  functions,  relationships,  and  dependencies  of 
constituent  systems,  both  internal  and  external.  This  includes  end-to-end 
functionality  and  data  flow  as  well  as  communications.  The  architecture  of  the 
SoS  provides  the  technical  framework  for  assessing  changes  needed  in 
systems  or  other  options  for  addressing  requirements. 

As  can  be  seen  by  the  above,  architectural  definition  is  a  key,  if  not  the  key,  element 
in  defining  and  controlling  the  development  of  an  SoS.  Although  a  non-trivial  task,  the 
development  of  the  architectures  for  the  SoS  provides  the  basis  for  evaluation  of  the  SoS. 

SoS  practitioners  have  started  to  look  at  how  to  use  various  architectures  for  optimizing 
performance  over  various  high-level  capability-based  metrics  using  approaches  such  as 
process  modeling  (Osmundson  &  Huynh,  2005;  Bindi  et  al.,  2008)  and  multi-attribute  utility 
theory  (Morrice,  Butler,  &  Mullarkey,  1998).  Additionally,  analysis  has  been  conducted  on 
methodologies  that  can  support  the  conduct  of  cost/performance  trades  for  the  SoS  (Luman, 
2000)  based  on  requirements  allocations.  Although  these  methods  all  show  potential  for 
assisting  the  SoS  decision-maker,  they  tend  to  be  point-specific  evaluation  tools  and  do  not 
appear  to  provide  the  decision-maker  with  a  tracking  and  predictive  capability  method 
similar  to  the  technical  performance  measure  (TPM)  used  at  the  system  level  for  predicting 
capability  during  development.  In  part,  this  is  because  the  SoS  requirements  decomposition 
process  is  not  a  well-understood  practice,  as  noted  by  participants  in  a  U.S  Army  Workshop 
on  exploring  enterprise,  system  of  systems,  and  system  and  software  architectures  (Bergey, 
Blanchette,  Clements,  &  Klein,  2009).  However,  such  an  SoS  performance  prediction 
methodology  is  needed  as  the  demand  for  predicting  capability  during  SoS  development  is  a 
constant  data  call  from  management. 

Historically,  M&S  has  often  been  used  in  the  early  stages  of  system  development  to 
aid  with  performance  prediction  of  the  system  with  the  anticipation  that  effective  M&S  can 
reduce  programmatic  risk  and  cost.  However,  the  extension  of  M&S  into  an  SoS  application 
is  challenging.  If  not  done  in  a  way  that  accurately  represents  the  status  of  the  individual 
systems’  capabilities,  the  SoS  interfaces,  and  the  planned  operational  employment  can 
result  in  erroneous  outputs.  This  is  driven  by  the  need  of  the  constituent  system  program 
managers  to  focus  their  M&S  efforts  on  activities  directly  related  to  their  individual 
programmatic  needs.  Furthermore,  the  lack  of  an  institutional  mandate  for  interoperable 
M&S  tools  means  that  this  SoS-specific  need  is  often  not  funded  and  may  not  be  perceived 
as  a  benefit  to  the  constituent  system  (Oswalt  et  al.,  201 1 ).  Even  if  desired,  as  noted  by 
Erhardt,  Flanigan,  and  Herdklick  (2010),  “the  development  of  a  federated  model 
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representing  the  SoS  can  also  be  expensive  and  time  consuming.”  Erhardt  et  al.  (2010) 
noted  that  for  accurate  representation  of  the  SoS  performance,  there  existed  the  need  for 
the  modeling  to  account  for  multiple  aspects  of  the  design  at  all  stages  of  development, 
including  the  accurate  representation  of  the  constituent  systems  within  the  SoS,  the 
integration  method  used  to  create  the  SoS,  the  operating  environment,  and  the  physical 
environment.  Even  if  this  data  were  available,  Erhardt  et  al.  (2010)  noted  that  there  would 
exist  a  risk  that  integrations  of  the  system  level  models  into  an  SoS  model  could  result  in 
erroneous  data  outputs  due  to  difference  in  fidelity  among  the  system  level  models  and 
issues  such  as  differing  baseline  assumptions  and  methods  of  dealing  with  rounding  error. 
Therefore,  physics-based  M&S  as  a  tool  for  use  in  predicting  the  performance  of  an  SoS 
may  not  be  the  tool  that  it  is  often  envisioned  as  an  aid  to  the  decision-maker. 

For  standalone  systems,  TPMs  are  often  used  during  development  as  a  leading 
indicator,  providing  the  PM  with  confidence  that  the  configuration  end  item  will  meet  its 
stated  required  capabilities.  The  concept  of  TPMs  was  originally  developed  by  the  Program 
Executive  Office  for  Air  ASW,  Assault,  and  Special  Mission  Programs  during  the  mid-90s 
(Pisano,  1995)  in  response  to  the  identification  of  the  existence  of  a  gap  in  information  from 
earned  value  data  that  was  not  tied  to  technical  performance.  The  International  Council  on 
Systems  Engineering  (INCOSE)  has  published  a  technical  measurement  guide  (Roedler  & 

Jones,  2005)  that  lays  out  the  standard  acquisition  approach  for  relating  operator 
requirements  to  quantifiable  and  measureable  data  that  can  be  obtained  during 
development.  Traditionally,  performance  expectations  fora  developmental  program  are 
established  by  defining  a  set  of  criteria  called  the  measures  of  effectiveness  (MOEs).  MOEs 
are  generally  used  to  define  the  level  of  operational  success  desired  of  the  developing 
capability  that  is  related  to  the  mission  and  environment  for  which  the  system  is  being 
developed  and  represent  the  end  users’  desires  for  system  capability  in  terms  of  operational 
value.  Critical  elements  of  the  MOEs  generally  get  translated  into  the  key  performance 
parameters  (KPPs)  of  a  system  that  are  used  to  help  drive  the  critical  performance  aspects 
of  a  proposed  design.  Verification  of  performance,  however,  generally  occurs  late  in  the 
program  development  cycle,  and  an  indicator  of  the  desired  performance  attainability  prior  to 
testing  is  desired  to  justify  the  ongoing  investment.  For  the  independent  system  level,  this  is 
obtained  through  the  selection  of  TPMs  that  represent  selected  physical  and  functional 
characteristics.  Trend  monitoring  of  the  various  TPMs  is  then  used  to  provide  the  PM  with 
confidence  that  the  system  should  eventually  deliver  its  required  capability.  This 
methodology  presents  challenges  when  applied  to  an  SoS  for  several  reasons.  First,  the 
SoS  PM  does  not  have  direct  control  over  the  developmental  PMs,  thus  gaining  and 
maintaining  status  of  the  individual  system  TPMs  may  not  be  achievable.  Second,  and 
perhaps  more  relevant,  is  that  even  if  the  data  is  obtainable,  it  may  not  be  relevant  with 
respect  to  how  the  constituent  systems  are  used  within  the  SoS.  MITRE  (Garvey  &  Cho, 

2005)  proposed  an  approach  to  resolving  this  issue  with  the  development  of  a  TPM  risk 
index  (TRI)  that  proposes  a  methodology  for  integrating  constituent  system  TPMs  into  a 
single  value  to  define  the  overall  performance  risk  for  the  SoS. 

Although  this  method  provides  insight  into  the  developmental  risk  of  a  SoS  based  on 
the  constituent  system  TPMs  within  each  system,  it  does  not  appear  to  account  for  several 
factors  that  may  influence  the  performance  of  an  acknowledged  SoS,  including  any  changes 
in  the  mission  scenarios  or  planned  operational  use  of  the  SoS  and  how  the  individual 
system’s  performance  may  be  modified  by  their  integration  within  an  SoS.  Alternatively,  for  a 
complex  SoS  (Jamshidi,  2006)  proposed  approach  to  determining  the  effectiveness  of  the 
SoS  that  treated  the  MOE  as  an  SoS-level  feature  and  MOPs  as  constituent-level 
components  with  TPMs  being  at  a  subsystem  level  (Jackson,  Sedrick,  &  Tayeb,  2009)  and 
proposed  the  incorporation  of  Measures  of  Suitability  (MOS)  as  a  measure  to  the  extent  to 
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which  the  system  integrated  with  the  operational  environment.  However,  the  proposed 
method  was  focused  on  determining  the  “best”  design  for  a  SoS,  considering  the  component 
capabilities  and  their  interactions  using  a  relaxed  multi-objective  optimization  process,  not 
as  a  performance  prediction/monitoring  tool. 

A  Missing  Link:  SoS  Performance  Prediction  Measures 

Historically,  system-level  programs  have  used  TPM  tracking  and  M&S  results  during 
development  to  predict  whether  a  system  will  meet  the  required  performance  objectives.  As 
noted  above  though,  the  use  of  system-level  M&S  and  TPMs  does  not  appear  to  be  directly 
translatable  to  SoS  due,  in  part,  to  the  inherent  complexity  of  the  SoS  and  the  operational 
flexibility  the  end  user  has  in  employing  the  SoS.  This  means  that  a  predictive  methodology 
for  developing  SoS  performance  is  needed.  The  methodology  needs  to  address  the  use  and 
integration  of  the  constituent  systems  into  the  SoS  and  the  decision-maker’s  and  user’s 
options  in  the  use  of  these  systems.  As  noted  in  the  SoS  Systems  Engineering  Guide 
(ODUSD[A&T]  SSE,  2008),  “the  SoS  systems  engineer  needs  to  establish  metrics  and 
methods  for  assessing  performance  of  the  SoS  capabilities  which  are  independent  of 
alternative  implementation  approaches.  A  part  of  effective  mission  capability  assessment  is 
to  identify  the  most  important  mission  threads  and  focus  the  assessment  effort  on  end-to- 
end  performance.”  This  is  especially  important,  as  one  of  the  key  challenges  a  PM  faces  is 
the  need  to  continually  defend  his  or  her  programs  in  terms  of  their  viability  against  not  only 
the  threats  for  which  the  system  was  designed  to  counter  at  program  initiation,  but  against 
any  emergent  threats  occurring  since  program  initiation.  This  viability  needs  to  be  shown  in 
terms  of  predicted  performance  during  development,  or  the  SoS  faces  an  increased  risk  of 
cancellation. 

Just  as  at  the  constituent  system  program  must  decide  on  what  attributes  are 
measureable  during  development,  the  SoS  PM  must  decide  what  technical  measures  need 
to  be  monitored  and  predicted  in  order  to  gain  insight  into  whether  the  SoS  will  yield  the 
required  performance.  In  seeking  to  address  this  need,  the  authors  proposed  a  deterministic 
SoS  performance  methodology  for  use  by  an  SoS  in  the  area  of  performance  prediction 
during  development  (Volkert,  Stracener,  &  Yu,  2012).  The  methodology,  summarized  in  the 
section  on  SoS  Performance  Measure  Development,  develops  a  single-point  SoS 
performance  measure  (SPM)  value  that  can  be  predicted  over  time  to  support  evaluation 
and  monitoring  of  an  SoS.  The  nature  of  SoS  development  means  that  this  metric  will  need 
to  operate  in  the  realm  of  imperfect  knowledge;  thus,  an  understanding  of  the  impact  of 
variability  on  each  of  the  input  values  will  be  required  to  understand  the  range  of  responses 
that  may  actually  be  present.  A  literature  review  was  conducted  to  verify  that  a  metric-based 
approach  was  an  appropriate  decision-making  aid  in  determining  what  analysis  was 
presently  missing  that  a  methodology  should  address  to  be  of  benefit.  It  was  determined  that 
the  developed  methodology  should 

•  begin  with  the  establishment  of  a  defined  mission  thread  and  it  mission  states; 

•  address  the  system  level  components  in  terms  of  performance,  sustainability, 
and  usage  within  each  state  of  the  mission;  and 

•  introduce  variability  into  the  calculations  to  understand  the  impact  of  multiple 
mission  scenarios. 

Having  defined  the  need  and  desired  capabilities  for  an  SoS  metric  equivalent  to  that  of  the 
TPM,  let’s  look  at  a  proposed  methodology  for  developing  the  SPM  value. 
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SoS  Performance  Measure  Development,  a  TPM  Equivalent 

For  an  SoS,  the  need  to  focus  on  end-to-end  performance  requires  that  the  SPM  be 
evaluated  across  the  range  of  potential  operations  (herewith  called  mission  states  [M])  that 

comprise  the  end-to-end  mission  ( M )  such  that  /W,c=  M  for  mission  states  /  =  1 , 2 . n, 

where  the  performance  of  the  each  mission  (P mission)  can  be  expressed  as  a  function  of  the 
level  of  performance  capabilities  of  the  mission  states  P mission  state(i)  (for  future  ease  referred  to 
as  PmS(i>),  which  can  in  turn  be  expressed  as  a  function  of  the  individual  performance 
capabilities  (Cs/,)  used  in  a  specific  mission  state  (Mi),  the  supportability  of  the  individual 
capabilities  (Ss,y)  within  that  mission  state,  and  the  operational  usage  of  the  individual 
capabilities  (Ukij)  within  that  mission  state,  or  mathematically  as  shown  in  Equation  1 . 

f3 mission  ~  d(Pms{X)'Pms{2)’  ’^rns{n))  0) 


where  PmS(i)  represents  the  performance  of  the  SoS  in  the  conduct  of  a  specific  mission  state 
/  and  j  represents  the  factors  associated  with  a  specific  capability  or  integrated  capability  for 

the  set  of  system  j  =  1,  2 . m  that  are  within  a  specific  mission  state.  This  hierarchical 

arrangement  is  shown  in  Figure  3.  Within  the  mission,  each  mission  state  must  be 
represented  by  a  unique  performance  value  such  that  the  characteristics  of  that  mission 
state  cannot  be  directly  compared  against  any  other  mission  state  within  the  mission.  Thus, 
we  can  derive  the  expression  of  performance  for  a  single  specific  mission  state,  as  shown  in 
Equation  2. 


n  ...  —  f.rrs  rs  .  rjh 

r  m5(i)  J  i  vWl'  ■" >  Wm'  u  il>  ■■■  >  u 


0,  fori  =  1,  2, 


(2) 
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Figure  3.  Hierarchical  Mapping  of  the  SoS  to  System-Level  Attributes 
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Let  us  now  look  at  the  functional  elements  ( Cs,y ,  SSij,  Ukjj)  within  each  mission  state 
and  how  these  values  may  be  determined. 

Mission  State  Parameters 

Let  Cfj  be  the  performance  of  an  individual  system  /within  mission  state  /'as  part  of 
the  SoS  as  a  time-specific  value  within  the  specific  mission.  The  factors  impacting  Cfj 
include  the  system-level  performance  capability,  Cfj,  and  how  that  capability  is  impacted  by 
its  inclusion  within  the  SoS  (either  due  to  a  different  operational  environment  or  the  impact 
of  integration  with  other  SoS  components).  Generically,  we  can  state  that  for  any  given 
system 

=  (otJ  x  (3) 

where  Cfj(t )  represents  the  performance  capability  of  the  system  j  as  a  standalone  system 
at  a  specific  period  of  time,  the  impact  of  its  inclusion  within  a  SoS  is  corrected  for  by  the 
adjustment  weight  uiij.  The  weighting  factor  u>ij  can  represent  either  an  increasing  or 
decreasing  impact  to  the  performance  capability  of  C®(t)  due  to  its  incorporation  into  the 
SoS.  The  challenge  to  the  practitioner  is  determining  how  to  evaluate  these  parameters 
within  the  SoS  and  with  the  limitations  in  data  that  may  be  available  in  determining  these 
values.  To  determine  the  system’s  expected  performance,  Cfj,  the  SoS  PM  could  either  use 
input  performance  data  (present  and  predicted)  from  the  standalone  capabilities  PM  or,  in 
the  absence  of  detailed  performance  data,  use  a  performance  growth  chart  based  on  the 
capabilities  expected  value  at  maturity  and  knowledge  of  its  present  developmental  status. 
One  method  of  determining  where  a  capability  stands  in  reference  to  its  projected  end  state 
performance  is  through  the  use  of  technology  s-curves.  Technology  s-curves  are  one 
method  used  to  portray  the  maturation  of  new  technology  as  it  progresses  from  concept  to 
onset  of  production,  through  rapid  growth,  to  the  onset  of  maturity  and  finally  maturity,  as 
shown  in  Figure  4. 


Time/SoS  Build  Variant 


Figure  4.  Notional  Technology  S-Curve  Mapped  to  Developmental  Events 
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For  capabilities  that  are  integrated  with  others  to  deliver  required  capability  within  a 
mission  state,  this  issue  becomes  more  complex.  Equation  3  can  still  be  used  for  an 
evaluation  of  the  impact  of  integration,  and  its  impact  on  maturity  should  occur.  One 
methodology  for  assisting  in  this  evaluation  already  exists,  which  the  practitioner  may  wish 
to  consider.  Specifically,  as  noted  within  the  SoS  Systems  Engineering  Guide  (ODUSD[A&T] 
SSE,  2008):  “A  part  of  effective  mission  capability  assessment  is  to  identify  the  most 
important  mission  threads  and  focus  the  assessment  effort  on  end-to-end  performance.” 

This  use  of  mission  threads  lends  itself  to  the  evaluation  of  the  status  of  integrated 
capabilities  using  a  systems  readiness  level  (SRL)  methodology  (Sauser,  Ramirez,  Henry,  & 
DiMarzio,  2008),  which  evaluates  the  impact  of  integrating  capabilities  based  upon  the 
individual  capabilities  linked  within  a  mission  thread  and  their  assessed  technology 
readiness  level  (Director,  Research  Directorate  [DRD]  Office  of  the  Director,  Defenes 
Research  and  Engineering  [DDR&E],  2009)  and  integration  readiness  level  (Sauser  et  al., 
2010). 

The  second  functional  element  S,y,  or  as  a  time  variant  factor  S/)(t),  can  be  shown  as 
seen  in  Equation  4,  which  reflects  the  integration  of  a  standalone  capability  sustainment 
approach  into  an  SoS.  Similar  to  the  performance  capability,  this  incorporation  may  result  in 
a  change  of  its  sustainability  performance  due  to  changes  in  how  the  capability  is  now  going 
to  be  sustained  within  the  SoS.  This  change  in  sustainment  can  result  in  increase  or 
decrease  to  the  system’s  reliability  and  operability  dependent  on  the  SoS  approach  to  these 
issues.  Thus,  the  impact  at  a  system  level  can  be  defined  as 

Sij(.t)  =  ^ j  x  (4) 

where  5®(t)  reflects  the  existing  level  of  sustainability  for  the  known  product  at  a  specific 
point  in  time,  and  /^reflects  the  weighting  value  applied  to  the  existing  sustainment  method 
of  the  baseline  capability  to  reflect  the  impact  of  incorporating  the  capability  into  the  SoS 
sustainment  philosophy.  In  the  early  stages  of  SoS  development,  understanding  this 
sustainment  delta  may  not  be  quantifiable  in  terms  of  impact,  but  as  the  SoS  develops  over 
time,  this  issue  should  become  clearer  to  the  PM.  In  general,  sustainability  for  a  system  is 
often  reflected  in  terms  of  the  products  reliability.  Thus,  if  the  SoS  PM  is  required  to  operate 
in  a  knowledge  gap,  the  reliability  bath  tub  curve  (see  Figure  5)  could  be  used  as  a  general 
methodology  by  a  PM  for  estimating  where  a  technology  stands  in  relation  to  its  end 
maturity  objectives  in  the  absence  of  other  data.  This  means  that  the  practitioner  could 
evaluate  where  the  specific  capability  is  with  respect  to  maturity  and  select  the  appropriate 
hazard  function  for  modeling. 
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Figure  5.  Reliability  Bathtub  Curve 

The  third  functional  element  Uij  is  designed  to  reflect  that  the  end  user  may  choose 
to  use  the  SoS  capabilities  applicable  to  a  specific  mission  state  in  multiple  ways  when 
executing  that  mission  state.  The  need  for  this  element  is  driven  by  the  inherent  flexibility 
that  exists  within  many  SoS  in  that  the  collective  capabilities  of  the  SoS  often  enable  it  to 
achieve  the  desired  performance  of  the  SoS  in  multiple  ways.  This  flexibility  is  generally 
viewed  as  a  positive,  as  it  enables  the  end  user  to  adjust  the  SoS  employment  to  deal  with 
operational  contingencies.  Indeed,  when  seeking  to  model  how  the  systems  composing  an 
SoS  will  be  used,  the  developmental  community  often  turns  to  the  user  community  to  solicit 
these  inputs.  For  defense  systems,  these  inputs  are  often  defined  in  terms  of  a  concept  of 
operations  (CONOPS)  document  that  indicates  how  the  delivered  capability  is  intended  to 
be  used.  For  the  SoS  engineer,  it  defines  the  need  to  be  able  to  evaluate  this  impact  on  the 
accomplishment  of  the  requirements  allocated  to  a  specific  mission  state  and  set  of 
capabilities.  For  this  analysis,  we  modeled  it  by  representing  the  utilization  of  the  capability. 

Thus  0  <  Uk/j  <  1  reflecting  the  use  of  capability  j  in  mission  state  /'  of  between  0  and  100%  of 
the  time.  Ukij  differs  from  the  other  two  elements  of  mission  state  performance  in  that  its 
value  is  generally  held  to  be  time  independent  within  the  specific  analysis  but  could  be 
variable  across  multiple  analyses.  This  reflects  that  for  any  specific  operational  condition, 
the  various  end  users  may  choose  to  employ  the  capabilities  in  different  combinations  to 
achieve  mission  objectives  resulting  in  a  /(-number  of  CONOPS  requiring  analysis  with 
respect  to  the  mission  state.  When  a  specific  operational  usage  scenario  and  set  of  usage 
rates  from  the  range  of  usage  options  is  applied  to  the  individual  systems  used  within  a 
mission  state,  the  value  of  Ukij  can  be  incorporated  into  the  system-level  determination  of 
performance  based  on  the  operator  inputs. 

Developing  a  Single  Point  SPM  Value  for  a  Mission  State 

As  shown  in  Figure  3,  each  system  within  a  mission  state  is  assumed  to  have 
performance  that  is  modeled  as  determined  by  the  status  of  the  input  variables  and 
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S-y(t)  as  defined  by  Equations  3  and  4.  For  the  simplicity  of  notation,  we  drop  the  time 
parameter  and  consider  only  the  random  variables  Cfj  and  5?-.  Thus,  at  any  specific  time  t 
the  performance  value,  independent  of  the  planned  system  usage,  can  be  defined  as 


C  =  Cs  x  Ss 

sys(ij)  y  ij 

where  Csys^  represents  the  combined  aspects  of  sustainability  and  performance  for  the 
individual  system  j  in  mission  state  /'. 


(5) 


The  next  step  of  the  methodology  incorporated  the  impact  of  the  operational  use  of 
the  various  systems.  This  allowed  us  to  develop  a  single  point  value  for  the  systems 
performance  at  any  given  time  within  the  mission  state  for  a  given  usage  value  of  the 
system  as  the  operator  specified.  For  this  example,  we  assumed  that  the  values  of  each 
capability  can  be  linearly  summed  to  produce  the  mission  state  performance  value  P</(f), 
referred  to  for  simplicity  as  P,y.  Therefore,  each  system  j  performance  in  mission  state  /  could 
be  defined  as 


P  =C  ....  xUk  =C..  xS" 

V  sys(y)  IJ  IJ  IJ 


xUk 


(6) 

for  that  set  of  system  capabilities  at  that  specific  point  in  time  and  for  the  specific  usage  rate 
represented  by  a  specific  Ukij. 


Let  us  now  expand  on  Equation  2  for  a  mission  state  where  the  systems  provide 
independent  levels  of  capability  designed  to  achieve  a  specific  level  of  performance. 
Although  it  is  recognized  that  a  range  of  values  could  contribute  to  determining  Csys®,  it  is 
assumed  that  at  any  specific  point  in  time  the  program  will  only  be  sequentially  evaluating 
single  sets  of  inputs.  Thus,  the  combined  performance  for  the  systems  for  any  single  set  of 
data  can  be  defined  by  summing  the  individual  system  performance  values  such  that 


P,m  =2 >  =  Z  CsysUj) X  (7‘  =  £  C|  x  Si  X  [/ * 

M  J=l  ./=!  (7) 

Using  Operational  Drivers  to  Develop  a  Range  of  SPM  Values 

With  the  elements  of  each  mission  state  now  defined,  and  with  a  methodology  of 
determining  a  single  value  for  a  mission  state  shown,  a  projection  of  performance  over  time 
and  the  development  of  tolerance  bands  similar  to  those  used  in  a  TPM  chart  can  now  be 
developed.  The  tolerance  boundaries  are  established  by  incorporating  the  fact  that  a  variety 
of  operational  employment  schemas  may  be  used  in  employing  the  SoS  and  its  collective 
capabilities.  We  can  model  this  by  stating  that  for  each  M )  there  is  associated  with  it  a 
specific  set  of  t/y,  values  reflecting  the  unique  CONOPS  modeled  within  the  specific 
analysis.  Thus,  for  any  designated  family  of  potential  CONOPS  relevant  to  the  SoS 
operations  within  the  mission  state  as  determined  by  the  user  community,  a  new  value,  as 
shown  in  Equation  8,  must  be  defined,  which  we  call  P(t)mS(i)Set,  for  ease  referred  to  as  Pms(i)set, 
reflects  the  averaged  impact  on  performance  of  all  potential  applicable  CONOPS  of  the  Mi 
combined  such  that 


P{t) 


ms  (i)  set 


o  o  m 

^jY  k^ms(i)  =  y,  7  k  ....j  U  ij 

k=\  k= 1  j= 1 


(8) 


where  for  each  mission  state  /'  there  exists  a  unique  set  of  t/y  values  such  for  each  PmS(i),  k 
reflects  the  maximum  number  of  CONOPS  modeled  and  that  y,  reflects  the  weight  given  to 
any  specific  set  of  (/j- values  such  that  £/,  =  1 .  This  single  point  value  is  what  we  refer  to  as 
the  mean  performance  value  for  the  mission  state.  The  value  may,  as  desired,  be 
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normalized  against  mission  state  the  KPP  requirements  and  effectively  represents  what  we 
are  classifying  as  the  SPM  value  for  the  mission  state.  Assuming  that  all  modeled  CONOPS 
support  development  of  the  required  level  of  performance,  the  upper  and  lower  tolerance 
bands  can  now  be  determined  using  Equations  9  and  10  for  the  SoS  by  reviewing  the 
individual  values  of  PmS(i)  for  each  unique  set  of  Ukij  values  and  using  the  maximum  and 
minimum  of  these  normalized  terms  as  the  upper  and  lower  tolerance  points  such  that 

^(0  ms(i)set(uppertolerance)  milX  J  1,2,..,  o}  ^g 

and 


ms(i)set(lowertolerance)  ni i II  {  Pm  ,  (  0  -  A  1,2,..,  o}  (10) 

for  the  set  of  values  at  time  ( t )  contributing  to  that  mission  set  and  for  all  operational  usage 
sets  evaluated.  The  determination  of  the  projected  values  of  SPM  over  time  for  a  mission 
state  is  thus  documentable  by  the  fact  that  the  individual  capability’s  within  each  mission 
state  are  considered  time  variant  for  performance  (t)  and  for  sustainability  S/)(t).  By 
determining  the  predicted  values  for  each  component  system  within  each  mission  state  over 
the  SoS  development  cycle  (U,  t2, ...,  tk),  a  time  variant  value  of  each  of  the  mission  states 
can  be  determined.  The  combination  of  the  mission  state  values  allows  for  a  single-mission 
performance  value  for  an  SoS  to  be  defined  and  plotted.  This  provides  that  the  SPM  curve 
then  has  the  potential  to  provide  the  PM  with  insight,  similar  to  the  traditional  TPM  chart  (see 
Figure  6). 


Figure  6.  TPM  to  SPM  Chart  Comparison 
Developing  a  Mission-Level  SPM  Value 

To  date,  the  SPM  metric  seems  to  provide  optimal  value  at  the  mission  state  level  as 
the  combination  of  data  to  a  single  non-dimensional  value  at  the  mission  level  starts  to 
significantly  remove  the  end  value  from  its  causes  and  diminishes  the  insight  provided  to  the 
SoS  PM.  As  such,  we  will  not  spend  much  time  in  this  paper  on  the  development  of  the 
mission-level  SPM  value.  Basically,  to  develop  the  mission-level  SPM  value,  the  mission 
state  values  need  to  be  combined.  Many  methods  exist  for  combining  data  from  various 
sources  and  types  dependent  on  the  data  type.  For  our  purposes,  because  the  intent  of 
SPM  is  to  use  it  as  a  leading  metric,  the  normalization  of  the  data  against  a  base  value  for 
each  mission  state  offers  potentially  the  easiest  and  most  meaningful  approach.  This 
approach  allowed  us  to  define  the  capabilities  within  a  mission  state  with  an  objective  value 
and  allows  the  mission  state  to  be  represented  by  a  value  against  which  present 
performance  can  then  be  measured,  and  against  which  progression  during  development  can 
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be  predicted.  This  approach  appears  to  be  in  line  with  existing  DoD  guidance  (ODUSD[A&T] 
SSE,  2008),  which  stated  that  for  SoS,  “SoS  requirements  are  often  cast  in  terms  of  broader 
capability  objectives.”  The  methodology  related  to  development  of  a  single  mission  value 
was  developed  and  documented  by  Volkert,  Stracener,  and  Yu  (2012). 

Having  now  laid  out  a  methodology  for  calculating  SPM,  let  us  use  an  example  to 
help  demonstrate  the  concept  using  a  mission  state  within  a  notional  detection-to- 
engagement  sequence  as  the  basis  for  demonstrating  this  deterministic  model. 

Mission  State  SPM  Use  in  a  Generic  Anti-Air  Mission  Example 

We  bound  the  limits  of  this  example  by  defining  a  notional  detect-to-engagement 
mission  that  is  combined  of  the  following  mission  states:  search,  detect,  classify,  track,  and 
attack/neutralization.  This  defines  our  mission  ( Pmission )  as  being  composed  of  n  =  5  mission 
states  ( PmS(i ))■  The  hierarchical  layout  of  the  mission  to  mission  state  of  interest  (track)  to  its 
component  capabilities  is  shown  in  Figure  7.  To  further  bound  the  example,  within  this  set  of 
mission  states  we  will  focus  on  the  development  of  the  SPM  chart  for  the  track  mission 
state.  Although  the  capabilities,  their  performance,  and  sustainability  varies  from  mission 
state  to  mission  state,  the  methodology  demonstrated  is  expected  to  be  tailorable  to  each 
mission  state  and  to  its  component  and  interfacing  systems. 


cs  ss 
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Figure  7.  Example  SoS  Mapped  to  Mission  States  and  Search  State  Systems 

Track  Mission  State  Capabilities  and  Analysis  Assumptions 

For  the  track  mission  state,  we  notionally  defined  the  desired  SoS  as  being 
composed  of  n  =  3  capabilities.  The  three  sensor  capabilities  selected  as  part  of  this  mission 
state  include  an  infrared  tracking  system,  an  airborne  radar,  and  a  ground-based  phased 
array  radar  that  all  provide  inputs  to  a  common  command  and  control  (C&C)  system  to 
which  they  are  being  interfaced  with.  The  systems  were  selected  to  reflect  the  capabilities 
that  would  be  representative  of  a  range  of  mature  systems  that  were  in  production  to  those 
under  development  or  would  require  significant  integration  with  other  capabilities.  All  values 
with  respect  to  performance,  CONOPS,  supportability,  and  so  forth,  were  chosen  purely  to 
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aid  in  the  demonstration  of  the  methodology  and  do  not  reflect  any  specific  system 
performance  parameters.  Table  1  provides  this  initial  mapping  of  systems  to  their  anticipated 
performance  and  adjustments  with  respect  to  being  in  the  SoS  (i.e.,  integrated  into  a 
common  C&C  system),  while  Table  2  provides  a  set  of  notional  operational  usage  rates 
reflective  of  operator  CONOPS.  The  next  stage  was  to  map  the  capabilities  performance 
improvement  over  time  and  determine  the  projected  end  value  when  adjusted  for  the 
weighting  values  reflecting  the  base  capabilities  inclusion  within  the  SoS.  Table  3  provides 
this  notional  developmental  mapping  of  performance/sustainment  ( CSij(t )  and  SSij{t))  over 
the  developmental  timeline.  Note  that  for  ease  of  calculation,  the  maximum  expected  base 
sustainment  was  arbitrarily  set  to  1 . 


Table  1.  Notional  Tacking  Mission  Stage  Capabilities 


Capability 

(integrated 

w/C&C) 

Max.  Base 
Capability 

(Cfj) 

Units 

Max.  Base 
Sustainability 

(Sfj) 

Performance 
Adjustment 
due  to  SoS 
Inclusion/ 
Integration 

(W,y ) 

Sustainability 
Adjustment  due 
to  SoS  inclusion 

(Pu) 

Infrared  Sensor 

300 

tracks/hr 

1 

0.6 

0.8 

Airborne  Radar 

500 

tracks/hr 

1 

0.8 

0.9 

Ground  Radar 

1000 

tracks/hr 

1 

0.9 

1.0 

Table  2.  Proposed  Tracking  Mission  State  System  Usage  Rates  (%) 


Capability 

CONOPS 1 

(W«) 

CONOPS  2 

(tA) 

CONOPS  3 

(iA) 

Infrared  Sensor 

20 

50 

20 

Airborne  Radar 

10 

10 

10 

Ground  Radar 

20 

10 

10 

C&C  System 

30 

25 

50 

Table  3.  SoS  Performance  and  Sustainability  Capability  Adjustment  at  Time  (t) 


Capability 

CsiAU) 

SS,y(fi) 

CSij(t2) 

Ssij(t2) 

CS,y(t3) 

SS,y(f3) 

CS,y(f4) 

SS,y(f4) 

Infrared  Sensor 

0.5 

0.2 

0.6 

0.4 

0.7 

0.6 

0.8 

0.8 

Airborne  Radar 

0.6 

0.5 

0.7 

0.6 

0.8 

0.7 

0.9 

0.8 

Ground  Radar 

1 

1 

1 

1 

1 

1 

1 

1 

For  this  example,  the  growth  scale  factor  was  taken  on  basically  a  linear  scale  if  the 
capability  was  deemed  developmental.  For  capabilities  that  were  viewed  as  “mature,”  the 
values  were  set  to  1 .  With  the  baseline  assumptions  and  growth  patterns  for  performance 
and  sustainability  defined,  we  could  calculate  mission  state  values  for  each  capability  with 
performance  represented  in  terms  of  target  tracks  maintained  per  capability  and 
sustainability  as  a  non-dimensional  value  at  each  specific  analysis  timeframe.  Table  4 
documents  these  results  as  shown  across  the  notional  developmental  timeline. 
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Table  4.  Calculated  Values  for  Growth  of  Performance  and  Sustainability  of 

Capabilities 


Capability 

Cs„(  fi) 

Ssn(  fi) 

Csn(t2) 

Ssn(t2) 

Csn(  t3) 

Ssii(h) 

CSn(U) 

SSn(U) 

Infrared  Sensor 

90 

0.16 

108 

0.32 

126 

0.48 

144 

0.64 

Airborne  Radar 

240 

0.45 

280 

0.54 

320 

0.63 

360 

0.72 

Ground  Radar 

900 

1 

900 

1 

900 

1 

900 

1 

Incorporation  of  CONOPS  Impact 


The  next  step  of  the  methodology  incorporated  the  impact  of  the  operational  use  of 
the  various  capabilities  as  defined  in  Table  2.  This  allowed  us  to  develop  the  value  for  the 
capabilities  performance  at  any  given  time  within  the  mission  state.  For  this  example,  we 
assumed  that  the  values  of  each  capability  can  be  linearly  summed  to  produce  the  mission 
state  performance  value  P,(f).  For  each  capability,  we  created  an  interim  calculation, 
Equation  1 1 ,  defined  as 


3(0  =  E  Cy(t)xS-j(t)xUy 

i= 1 


(11) 


which,  when  summed  across  the  n  =  3  capabilities  used  within  the  mission  state  yields  P,(f) 
for  that  set  of  capabilities  at  that  specific  point  in  time  and  that  specific  CONOPS 
represented  by  a  specific  UKij.  Table  5  summarizes  the  resulting  calculations  for  this 
example. 


Table  5.  Basic  Mission  State  Performance  Calculation  per  Implemented  CONOPS 


ti 

*3 

U 

Capability 

pm 

for 

«A) 

P4t) 

for 

«A) 

P4t) 
for 
(U3  ij) 

pm 

for 

«A) 

p*(t) 

for 

dA) 

P4t) 

for 

dA) 

P4t) 

for 

«A) 

P4t) 

for 

«A) 

P4t) 

for 

«A) 

P4t) 

for 

dA) 

P4t) 

for 

dA) 

P4t) 

for 

«A) 

Infrared 

2.9 

7.2 

2.9 

6.9 

17.3 

6.9 

12.1 

30.2 

12.1 

18.4 

46.1 

18.4 

Airborne 

10.8 

10.8 

10.8 

15.1 

15.1 

15.1 

20.2 

20.2 

20.2 

25.9 

25.9 

25.9 

Ground 

180.0 

90.0 

90.0 

180.0 

90.0 

90.0 

180.0 

90.0 

90.0 

180.0 

90.0 

90.0 

P4  (t)= 

193.7 

108.0 

103.7 

202.0 

122.4 

112.0 

212.3 

140.4 

122.3 

224.4 

162.0 

134.4 

The  results  of  this  data  could  now  be  plotted  into  a  growth  curve  of  SoS  performance 
over  time  against  which  test  and  developmental  data  could  be  applied  to  provide  the  SoS 
PM  with  insight  into  the  likelihood  of  meeting  performance  requirements.  Having  determined 
the  individual  values  of  performance  for  specific  CONOPS  associated  with  the  set  of 
capabilities  within  the  mission  state  over  time,  we  could  determine  the  average,  maximum, 
and  minimum  values  at  each  period  of  time  to  enable  the  creation  of  the  SPM  chart.  Table  6 
provides  a  summary  of  the  performance  values  for  the  search  mission  state  in  terms  of  the 
maximum,  minimum,  and  average  value. 


Table  6.  Calculated  Mission  State  Performance  Values 


U\ j 

U2*  j 

U\ i 

YPIk 

P min 

P max 

p4@  fi 

193.7 

108.0 

103.7 

135.1 

103.7 

193.7 

p4@  t2 

202.0 

122.4 

112.0 

145.5 

112.0 

202.0 

Pt@  fs 

212.3 

140.4 

122.3 

158.3 

122.3 

212.3 

p4@  f4 

224.4 

162.0 

134.4 

173.6 

134.4 

224.4 
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Incorporating  Uncertainty  Into  the  SoS  Performance  Measure  Metric 

So  far,  the  development  of  SPM  has  focused  on  a  deterministic  approach  to  the 
prediction  of  single-point  values  staged  sequentially  over  a  developmental  timeline  for  a 
specified  mission  set  where  the  input  values  were  deterministic.  However,  in  the  real  world, 
inputs  are  seldom  deterministic  in  nature,  and  not  all  operational  usage  concepts  are 
weighted  equally.  Although  the  above  approach  (Volkert  et  al.,  2012)  worked  to  develop  the 
SPM  metric  for  the  deterministic  state,  viewing  the  problem  stochastically  requires  that  the 
performance  be  rephrased  in  terms  of  its  value  and  probability  of  occurrence  at  each  stage. 
Let  us  now  briefly  discuss  how  uncertainty  could  be  introduced  into  the  determination  of 
SPM  values.  Basically,  we  must  answer  the  question  of  whether  we  can  define  the 
component  elements  of  the  function  and  their  variability  in  a  meaningful  way,  such  that  when 
the  variability  is  accounted  for,  the  SPM  metric  will  provide  an  improved  value  added  insight 
to  the  PM.  To  do  this,  we  would  need  to  reexamine  how  we  define  the  performance  of  a 
mission  as  being  composed  of  the  performance  of  a  set  of  defined  mission  states  (PmS(i))  that 
are  composed  of  individual  and/or  integrated  systems  functioning  as  independent 
capabilities. 

At  the  system  level,  we  could  expect  the  development  of  the  individual  systems  or 
their  integrated  capabilities  as  reflected  in  the  factors  Cfj(t)  to  vary,  as  few  systems  actually 
produced  the  exact  level  of  predicted  performance  expected  at  the  predicted  time.  This 
means  that  C/)(t)  would  be  expected  to  have  a  degree  of  variability  associated  with  it. 
Therefore,  we  will  need  to  represent  the  value  of  C/)(t )  as  an  effective  value,  denoted  as 
CfjS(t),  with  a  target  of  cfj(t ),  which  could  be  expressed  as  a  function  of  the  instance  value 
(cfj(t))  and  the  probability  of  having  such  an  instance  (P(cfj(t)  ))  such  that 

Cjf(t)  =f(c?j(t),P(c?j(t))  (12) 

Similarly  to  the  ability  to  predict  the  effective  value  of  Sfj(t),  denoted  as  Sfjs(t), 
for  the  SoS  based  on  the  individual  system  sustainment  plans  and  their  system 
sustainability  factor  will  of  necessity  be  an  imprecise  prediction  during  the  developmental 
stage  of  a  SoS  development  and  contain  an  element  of  variability 

Sff(t)  (13) 

Introducing  a  degree  of  variability  associated  with  C/)(t)  and  S-Ct)  means  that  we 
need  to  account  for  that  variability  in  the  combined  value  Csys®.  This  issue  can  become 
complicated  because  the  maturation  of  sustainment  is  often  independent  of  the  maturation 
of  performance  during  the  developmental  effort.  However,  as  the  SoS  design  matures  and 
proceeds  into  production  and  fielding,  sustainment  and  performance  often  become  closely 
linked. 


For  the  usage  equation,  we  needed  to  look  at  how  the  operators  view  the  system 
and  how  that  introduces  variability.  Generally,  when  modeling  how  the  systems  composing  a 
SoS  are  used,  a  survey  of  potential  users  with  respect  to  potential  operational  usage 
concepts  that  may  be  employed  by  the  operators  of  the  systems  within  a  given  mission  state 
is  taken.  The  users  then  specify  a  range  of  values  for  the  usage  of  each  system  reflecting 
their  personal  experiences  and  assumptions.  As  with  the  values  for  sustainment  and 
performance,  operational  usage  is  seldom  achieved  to  the  degree  of  accuracy  defined  in  the 
initial  plan  by  the  user  community.  This  means  that  U-j  should  also  be  expected  to  have  a 
degree  of  variability  associated  with  it  for  any  discrete  system  usage  set  k,  and  its  effective 
value,  denoted  as  Uff ,  can  be  expressed  as 
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(14) 


Uijk  =  / (uij,  P k(ufj)) 

where  Pk(ufj)  represents  the  probability  of  occurrence  of  event  ujj  for  any  specific  usage 
value  ufj.  Unique  to  the  usage  value  is  the  question  of  how  to  combine  the  independent 
values  into  a  single  representative  value  that  can  represent  the  overall  probability  range  of 
the  user  community.  Fortunately,  the  issue  of  how  to  combine  experts’  probability 
distributions  has  been  extensively  studied.  Clemen  and  Winkler  (1999)  reviewed  and 
summarized  a  variety  of  issues  inherent  in  mathematical  and  behavioral-based  approaches 
that  need  to  be  considered  in  developing  a  combinational  process  for  use  in  practice.  Their 
analysis  tended  to  indicate  that  mathematical  methods  performed  better  than  behavioral 
methods.  In  addition,  it  indicated  that  although  more  complex  combinational  methods  may 
be  more  accurate,  they  can  be  somewhat  sensitive,  and  simple  combinational  methods  tend 
to  perform  well.  Because  the  purpose  of  the  SPM  metric  is  to  provide  the  PM  with  a  simple 
tool  for  obtaining  insight  into  the  likelihood  of  the  SoS  performing  as  desired  upon 
completion,  the  approach  of  using  a  simplified  tool  for  correlating  users’  inputs  into  a  single 
value  and  a  representation  of  their  probability  of  achieving  that  value  may  be  a  usable 
approach.  Supporting  this  approach,  a  combinational  method  called  the  linear  opinion  pool 
(Stone,  1961)  will  be  looked  at  as  an  option  for  supporting  the  combination  of  the 
probabilities. 

Once  an  acceptable  method  is  determined,  and  accepting  that  the  usage 
determination  is  independent  of  the  combined  performance  and  sustainability  vectors,  a 
method  of  determining  the  system  level  variability  for  each  system  within  the  mission  state 
can  then  be  developed  to  generate  a  value  of  P(pSyS(ij ))>  which  will  represent  the  overall 
probability  of  achieving  the  desired  level  of  system  performance  for  system  j  within  a  given 
mission  state  /  using  subject  matter  expert  input  on  system  usage  options  for  the  averaged 
usage  of  the  system  in  that  mission  state.  If  we  view  the  individual  system  contributions  as 
independent  events  within  each  operational  usage  concept,  the  probability  for  the  specific 
mission  state  usage  scenario  of  m  systems  achieving  that  performance  could  then 
potentially  be  developed  and  expressed  by  P(pmS(i))-  This  would  allow  us  to  then  represent 
the  mission  state’s  combined  probability  of  achieving  the  predicted  performance  level;  and 
assuming  that  the  usage  of  capabilities  within  a  mission  state  are  mutually  exclusive,  we 
could  derive  the  variability  in  the  SPM  value  for  the  mission,  defined  as  P(pm<i))-  Developing 
the  analysis  and  methodology  supporting  this  determination  is  the  planned  subject  of  the 
authors’  future  research. 

Conclusion 

This  paper  focused  on  developing  a  technical  performance  methodology  applicable 
to  the  acknowledged  SoS.  The  SoS  PM  is  developed  as  a  leading  indicator  to  assist  the 
SoS  PM  by  providing  insight  into  the  risks  related  to  the  ability  to  obtain  desired 
performance.  First,  a  deterministic  method  was  developed  to  define  the  concept  and 
approach.  Using  a  generic  mission,  specifically  a  tracking  mission  state  as  an  example,  the 
SPM  methodology  was  then  demonstrated.  The  data  developed  indicates  that  the 
methodology  does  provide  the  SoS  PM  with  the  opportunity  to  gain  additional  insight  into  the 
ability  to  achieve  the  desired  performance  for  the  SoS,  without  the  need  for  intensive 
modeling  and  simulation.  This  insight  would  assist  the  SoS  PM  in  better  understanding  the 
degree  of  risk  he  or  she  faces  as  developmental  test  data  becomes  available.  Additionally,  it 
could  serve  as  a  tool  for  quickly  understanding  the  impact  of  CONOP  changes  and/or  the 
impact  of  new  capabilities  (threat  or  constituent  systems)  on  SoS  performance.  The  authors 
introduced  the  need  for,  and  proposed  approach  to,  extending  the  methodology  into  a 
stochastic  method  to  introduce  the  real-world  impact  of  variability  into  the  analysis  as  an 
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area  of  follow-on  study.  In  summary,  the  SPM  metric  seems  to  provide  value  for  a  PM, 
optimally  at  the  mission  state  level,  as  the  combination  of  data  to  a  single  non-dimensional 
value  at  the  mission  level  starts  to  significantly  remove  the  end  value  from  its  causes, 
diminishing  the  insight  provided  to  the  SoS  PM.  It  is  suggested  that  further  research  on  this 
area  and  validation  of  the  approach  against  real-world  case  studies  would  be  of  value  to  the 
SoS  management  community. 
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Growth  of  System  Complexity 


Growth  of  Systems  Complexity  into  the  Systems 
Century 


Growth  of 
the  Internet 


Growth  of  Electronics 
&  Computers 

Growth  of  Space 


Growth  of  Aviation 
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Why  shift  from  system  based  to  capability  based 
acquisitions  ? 


Decreasing  Funding 
to  support  Defense 
Investment  in 
increasingly 
expensive  systems 


1*S0  1*8E  1990  199E  £000  200E 


Decreasing  Cost  for 
Adversaries  to 
counter  traditional 
areas  of  military 
dominance 


Zephyr  Unmanned  Aerial  Vehicle  System 
$9,500 


Add  to  Cart 


The  Zephyr  UAV  system  is  a  complete  solution 
for  aerial  surveillance,  photography  or  as  a 
research  platform. 


fc  Ground  Control  Software 
•  Joystick  Controler  For  Pan  Tilt 
Camera  System 

fc  RC  controller  for  manual  flight 


Blackberry 
Secure  SIP  Client 


2410  ;01?  2020  20  £S  £010 


intensifies  the  desire  to  obtain  greater  utility  from  systems  developed  &  procured 
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New  Paradigm  of  System  of  System  Acquisition 


A  SoS  is  defined  as  a  set  or  arrangement  of  systems  that  results  when 
independent  and  useful  systems  are  integrated  into  a  larger  system  that  delivers 
unique  capabilities  [DoD,  2004(1)]. 


Type 

Definition  1 

Virtual 

Virtual  SoS  lack  a  central  management  authority  and  a  centrally  agreed  upon  purpose  for  the  SoS. 
Large-scale  behavior  emerges — and  may  be  desirable — but  this  type  of  SoS  should  rely  upon  relatively 
invisible  mechanisms  to  maintain  it. 

Collaborative 

In  collaborative  SoS,  the  component  systems  interact  more  or  less  voluntarily  to  fulfill  agreed  upon 
central  purposes.  The  Internet  is  a  collaborative  system.  The  Internet  Engineering  Task  Force  works  out 
standards  but  has  no  power  to  enforce  them.  The  central  players  collectively  decide  how  to  provide  or 
deny  service,  thereby  providing  some  means  of  enforcing  and  maintaining  standards. 

Acknowledged 

Acknowledged  SoS  have  recognized  objectives,  a  designated  manager,  and  resources  for  the  SoS; 
however,  the  constituent  systems  retain  their  independent  ownership,  objectives,  funding,  and 
development  and  sustainment  approaches.  Changes  in  the  systems  are  based  on  collaboration  between 
the  SoS  and  the  system. 

Directed 

Directed  SoS  are  those  in  which  the  integrated  SoS  is  built  and  managed  to  fulfill  specific  purposes.  It  is 
centrally  managed  during  long-term  operation  to  continue  to  fulfill  those  purposes  as  well  as  any  new 
ones  the  system  owners  might  wish  to  address.  The  component  systems  maintain  an  ability  to  operate 
indenendenflv.  blit  their  normal  onerational  mode  is  subordinated  to  the  eentral  managed  nnrnose 

The  field  of  SoS  engineering,  development,  integration,  sustainment,  and  management  requires  the  decision 
maker  to  face  both  the  traditional  challenges  associated  with  any  complex  system  (Jamshidi,  2006)  and  the 
additional  challenges  associated  with  having  to  analyze,  organize,  and  integrate  the  constituent  systems 

(existing  and  developmental)  into  an  integrated  SoS  capability. 
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System  of  Systems  Challenges 


•  SoS  acquisition  management  -  a  significant  increase  in 
complexity  over  traditional  system  acquisition 

•  Development  requires  that  significant  numbers  of 
technologies  be  integrated  to  one  another 

•  Challenges  traditional  development  monitoring  tools  and  cost 
models 

-  need  to  capture  integration  complexity 

-  level  of  effort  required  to  connect  individual  components 

•  Unintended  Consequences  -  high  degree  of  inter-linkage 
between  components  can  cause  unintended  impacts  to 
overall  system  performance 

-  components  are  modified  from  original  use 

-  Technology  change:  replaced  throughout  the  system  life  cycle 

“Large,  complex  systems  development  has  always  been  challenging,  even  when 
the  “only”  thing  a  program  manager  had  to  worry  about  were  cost,  schedule,  and 
performance  within  a  single  program”.  Smith  and  Meyers  (2008), 
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System  of  Systems  Tools-  an  emerging  field  of  study 


Technical  Risk  Index  (TRI)/Generalized 
Performance  Risk  Measure.  The  index 
shows  the  degree  of  performance  risk 
presently  in  the  SoS,  supports  identifying 
risk-driving  TPMs,  and  can  reveal  where 
management  should  focus  on  improving 
technical  performance  and,  thereby,  lessen 
risk. 1 


TRI  Value 
II - 


i  i  r 

Top  Curve:  TRIV  A 
Middle  Curve:  TRI,, 


Region  of  Unacceptable 
Performance  Risk 
(TRI  >  0) 


At  t6  All  TPMs 
►  Reach  Thresholds 


Figure  1.  System  Concept  Diagram 


t2  t3  t4 
Measurement  Date 

Figure  4.  Illustrative  TPM  Risk  Index  Time  History  Trend 


TRJ  =  0.483  TRI  =  0646  TRI  =  0303  TRI  =  0.472  TRI  =  0.728 

Figure  6.  An  Illustrative  SoS  Hierarchy,  TRI  Values,  and  Associated  Colors 


Process  Modeling.  A 

methodology  for  performing 
architectural  analyses  of 
complex  systems  of 
systems. 2 

System  Earned  Readiness  Management  (SERM).  A  monitoring  and 
evaluation  tool  for  the  planning  and  monitoring  the  progress  of  the 
system  development  effort  using  SRL  (combination  of  TRL  and  IRL) 
combined  with  the  prescribed  strategy  for  developing  the  SoS  and  an 
appropriately  constrained  optimization  model  to  formulate  the  optimal 
development  plan.3 


Forward  DwployFoico 


LAUNCHING  AREA 


kjraawgi 


Jjg- 

SjSp 

Figure  2.  Top-level  \iew  of  the  EXTEND™  Expeditionaiy  Waifare  Model  for  Sea  Basing. 


1  Garvey,  P.  and  Cho,  C.  (2005).  “An  Index  to  Measure  and  Monitor  a  System -of-Systems’  Performance  Risk”,  MITRE  Technical  Paper 

2  Osmundson,  J.,  and  Huynh,  T.,  (2005),  ‘A  Systems  Engineering  Methodology  for  Analyzing  System  of  Systems’,  Proceedings  of  the  1st  Annual  Systems  of  Systems  Engineering  Conference 

3  Sauser,  B.,  Ramirez-Marquez,  J.,  Magnaye,  R.,  (2009),  “Using  a  System  Maturity  Scale  to  Monitor  and  Evaluate  the  Development  of  Systems”,  Proceedings  of  the  6th  Annual  NPS  Acquisition 
Symposium 
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Performance  Prediction  within  a  SoS  - 

the  need  for  a  TPM  Equivalent 


Technical  Performance  Measurements1’2 

provide  insight  as  to  the  parameters  of  the 
specific  design  elements  of  the  system  and 
are  used  by  project  management  to  define  the 
measures  of  performance  and  acceptable 
variables  during  project  implementation. 


Technical  Performance  Measurement  values 
are  implemented  at  the  beginning  of  a  project, 
as  to  ensure  that  projected  performance 
values,  within  tolerable  variance  ranges,  are 
met.  Throughout  the  project,  the  actual 
performance  is  tracked  and  compared  by 
project  management  to  the  Technical 
Performance  Measurement  that  was  deemed 
acceptable  at  the  project’s  outset  -pmbok  (online 

4/17/12) 

An  equivalent  metric  for  SoS’s  does  not  seem  to  exist 


Time 


1  Pisano,  N.,  (1995),  “Technical  Performance  Measurement  Earned  Value,  and  Risk  Management:  An  Integrated  Diagnostic  Tool  for  Program  Management”,  Paper  presented  at  Defense  Acquisition  University  Acquisition 
Research  Symposium., 

2  Roedler,  G.  and  Jones,  C.  (2005)  ‘Technical  measurement’,  A  collaborative  project  of  PSM,  INCOSE,  and  industry,  ver.  1.0,  INCOSE  Technical  Report  No.  INCOSETP-2003-  020-01. 
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Development  of  the  SoS  Performance  Measure  (SPM) 
Metric 


the  SoS  systems  engineer  needs  to  establish  metrics  and  methods 
for  assessing  performance  of  the  SoS  capabilities  which  are 
independent  of  alternative  implementation  approaches.  A  part  of 
effective  mission  capability  assessment  is  to  identify  the  most 
important  mission  threads  and  focus  the  assessment  effort  on  end-to- 
end  performance.  1 


SffiSMU 


What  can  the  SoS  PM  know, 
system  of  system  level? 


Define  the  SoS  hierarchy  and  its 
perceived  application  to  fulfilling  the 
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What  can  the  SoS  PM  know,  obtain,  or  predict  at  the 
system  level? 


Performance  and  schedules  of  known  systems, 
predictions  for  those  under  development  or  where 
knowledge  is  limited.  Input  from  the  potential  users 


Figure  4:  Notional  Technology  S-Curve  mapped 
to  Developmental  Events 
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Figure  5:  Reliability  Bathtub  Curve 


“Nothing  succeeds  in  war  except  in 
consequence  of  a  well  prepared  plan.' 

Napoleon 


Putting  it  together  into  a  generic  methodology 


Define  the  mission  as  a  function  of  its  mission  stages 


Decompose  the  mission  stages  to  their  constituent  elements 


Adjust  the  constituent  elements  for  inclusion  in  a  SoS 

Incorporate  system  usage  within  SoS  and  determine 
system  level  performance  within  a  mission  state 

Determine  the  performance  level  of  the  mission  state  for  a 
specific  usage  scenario 

Adjust  for  the  range  of  usage  scenarios  being  evaluated 
To  determine  the  mean  value 


O 


P mission  Pms(2)>  •••  >  ^ ms(n )) 
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An  example  application-  establish  the  hierarchy 


Cs  Ss 

^  41'  J  41 


Cs  Ss 

42'  J  42 
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An  example  application-  define  the  parameters 


Detect  to  Engage  SoS 
Mission 
Performance 
Pm 


Search  State 

^ms(i)  =1 


Capability 

(integrated 

w/C&C) 

Max.  Base 

Capability 

Units 

Max.  Base 

Sustainability 

Performance 

Adjustment  due 
to  SoS  Inclusion/ 
Integration  (a)u ) 

Sustainability 
Adjustment  due  to 
SoS  inclusion 

Infrared  Sensor 

300 

tracks/hr 

1 

0.6 

0.8 

Airborne  Radar 

500 

tracks/hr 

1 

0.8 

0.9 

Ground  Radar 

1000 

tracks/hr 

1 

0.9 

1.0 

Classify  State 

^ms(i)  =3 


Track  State 

p 

1  ms(i)  =4 


Ground 

Radar 


Infrared 

Tracker 


uK, 


Attack  State 

^ms(i)  =5 


Detect  State 

/ 

Airborne 

Pms(i)  =2 

Radar 

rs  Ss 

^  41'  J  41 


Cs  Ss 

^  42'  J  42 


rs  Ss 

^  43'  J  43 


Capability 

CONOPS  1 

(U1 4.) 

CONOPS  2 

(V2 4i) 

CONOPS  3 

(U3 4.) 

Infrared 

Sensor 

20 

50 

20 

Airborne 

Radar 

10 

10 

10 

Ground 

Radar 

20 

10 

10 

C&C 

System 

30 

25 

50 

SMU  LYLE  v 

SCHOOL  OF  ENGINEERING  TSalST 


Rich  Volkert  - 13 


An  example  application-  calculate  performance  @ 
system  level 


Capability 

CsS0 

SSM 

csSi) 

<%) 

sMh) 

CVD 

Infrared  Sensor 

0.5 

0.2 

0.6 

0.4 

0.7 

0.6 

0.8 

0.8 

Airborne  Radar 

0.6 

0.5 

0.7 

0.6 

0.8 

0.7 

0.9 

0.8 

Ground  Radar 

1 

1 

1 

1 

1 

1 

1 

1 

Capability 

CSn(U) 

ssM 

Csitt 2) 

Cs,v(t3) 

csM 

Infrared 

Sensor 

90 

0.16 

108 

0.32 

126 

0.48 

144 

0.64 

Airborne 

Radar 

240 

0.45 

280 

0.54 

320 

0.63 

360 

0.72 

Ground 

Radar 

900 

1 

900 

1 

900 

1 

900 

1 

h 

h 

h 

t4 

Capab 

ility 

PS ) 

for 

(EAj) 

PS) 

for 

iu2S 

PS) 

for 

(U\ ,) 

PS) 

for 

PS) 

for 

(U2  n) 

PS) 

for 

(U\) 

PS) 

for 

(EAi> 

PS) 

for 

iu2S 

PS) 

for 

(^5ii) 

PS) 

for 

(EAj) 

PS) 

for 

(U2  n) 

PS) 

for 

(U\  j) 

Infrar 

ed 

2.9 

7.2 

2.9 

6.9 

17.3 

6.9 

12.1 

30.2 

12.1 

18.4 

46.1 

18.4 

Airbo 

rne 

10.8 

10.8 

10.8 

15.1 

15.1 

15.1 

20.2 

20.2 

20.2 

25.9 

25.9 

25.9 

Groun 

d 

180.0 

90.0 

90.0 

180.0 

90.0 

90.0 

180.0 

90.0 

90.0 

180.0 

90.0 

90.0 

p4 

(t)= 

193.7 

108.0 

103.7 

202.0 

122.4 

112.0 

212.3 

140.4 

122.3 

224.4 

162.0 

134.4 
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An  example  application-  calculate  performance  @ 
mission  system  level  &  draft  SPM  chart 


V'm 

U2m 

2P/k 

P  . 

min 

P 

max 

p4@  tt 

193.7 

108.0 

103.7 

135.1 

103.7 

193.7 

p4@t2 

202.0 

122.4 

112.0 

145.5 

112.0 

202.0 

p4@t3 

212.3 

140.4 

122.3 

158.3 

122.3 

212.3 

p4@t4 

224.4 

162.0 

134.4 

173.6 

134.4 

224.4 

Upper  tolerance  value 

Pit)  = 

V  /  ms{i)set{uppertoleunce) 


Mission  State 
Planned 
Performance 
Profile 

Mission  State 
Performance 
Threshold 


Lower  tolerance  value 

)  ms(i)set(lowertoleance)  =  vcm{Pms{i){t\k  =  \,2,..o) 


«V=  Ir,/(^(i)(  0)  =  IIr,g(c  ,t/‘s) 

fc=i  y-i 


Jfc=l 


Event  1  Event  2  Event  n 
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Incorporating  variability  to  represent  the  real  world  - 

the  next  step  forward 


Need  to  realize  that  predictions  with  respect  to  achievement  of  performance, 
usage  of  systems,  etc.  are  seldom  met  exactly.  How  do  we  deal  with  this  variability? 


The  starting  point,  incorporating 
a  probability  function  for  the 
system  drivers 


U$k  =  Pfc(ufy)) 


Stochastic  Deterministic 


'putune  fiafeen 
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Conclusion 


•  System  of  Systems  Development  are  an  increasing  trend  within  Defense  & 
pose  significant  challenges  in  management  &  performance 
prediction/monitoring 

•  The  development  of  new  tools  and  metrics  for  SoS  is  an  ongoing  field  of 
research  but  seems  to  be  missing  a  TPM  equivalent  for  a  SoS 

•  The  SoS  Performance  Metric  (SPM)  is  a  potential  tool  for  addressing  this 
area 

•  Potential  to  operate  within  the  constrained  knowledge  that  a  SoS  PM  may  have 

•  Expansion  of  the  tool  to  account  for  variability  should  enhance  its  usability  in 
quantifying  the  risk  to  achieving  performance 

•  Requires  verification  against  real  world  data 
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Back  Up  Material 
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